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Four  Principles  for  Designing  Instructions 
ABSTRACT 

This  paper  gives  four  principles  for  preparing  multimedia  instructional 
sequences,  and,  where  necessary,  the  experimental  methods  for  applying  the 
principles  successfully.  It  also  describes  the  empirical  experiments  on  which 
the  principles  are  based.  Principle  One  Is  a  criterion  for  good  terminology  for 
unfamiliar  objects,  actions,  and  situations,  with  methods  for  deriving  such 
terminology.  Principle  Two  tells  how  to  overlap  visual  and  spoken  elements  In 
time  (as  In  a  movie  or  lecture  with  slides)  In  order  for  good  associations  to  be 
formed.  Principle  Three  states  that  division  of  Instructions  Into  conceptual 
units  should  be  In  agreement  with  people's  natural  conceptualization.  Here,  a 
method  Is  presented  for  finding  the  natural  conceptualization.  Finally, 
Principle  Four  regards  mixing  audiovisual  Instruction  with  hands-on  practice  In 
learning  a  procedure.  These  principles  should  be  useful  In  a  variety  of 
situations. 


Four  Principles  for  Designing  Instructions 

Introduction 

This  article  contains  four  principles  for  designing  multimedia 
Instructions.  By  multimedia  Is  meant  visual  and  verbal  material  (such  as  a  film 
or  a  text  with  Illustrations)  apd  actual  practice.  The  Instructions  we  have 
focussed  on  are  for  assembly  of  physical  objects,  but  the  principles  are  not 
restricted  to  application  only  In  assembly. 

The  first  principle  deals  with  how  to  construct  terminology  for  use  with 
unfamiliar  objects,  actions,  or  situations.  The  second  principle  Is  how  to 
overlap  visual  and  spoken  material  In  time,  In  order  for  good  associations  to  be 
made.  The  third  principle  tells  how  to  divide  Instructional  material  into 
conceptual  units.  And  the  fourth  deals  with  mixing  audiovisual  Instruction  with 
hands-on  practice. 

Principles  one  through  four  are  In  general  both  task-  and 
subject-dependent.  For  example,  the  right  terminology  depends  on  the  task  or 
the  material  presented,  and  on  the  verbal  abilities  of  the  subjects.  The  amount 
of  hands-on  practice  could  depend  on  subjects'  manual  dexterity  and  experience 
with  similar  kinds  of  tasks. 

For  the  principles  which  are  subject-  or  task -dependent,  we  present  here 
the  experimental  methods  which  one  can  use  to  determine  subject  and  task 
parameters.  For  example.  In  Part  I  we  present  an  experimental  method  of  how  to 
develop  terminology  which  Is  adequate  for  a  task  and  for  subjects  who  will 
perform  the  task.  In  Part  III  we  present  an  experimental  method  for  the 
division  of  material  Into  conceptual  units.  It  Is  again  task-  and 
subject-dependent. 

In  some  cases  we  can  suggest  general  principles,  namely,  specific  do's  and 
don't's  that  should  apply  to  any  task  and  any  group  of  subjects.  For  example. 


in  Part  II,  a  visual  presentation  should  precede  or  be  In  synchrony  with  the 
related  spoken  presentation,  and  not  follow  it.  The  general  principles  which  we 
present  have  been  derived  from  empirical  experiments,  or  are  consistent  with 
what  we  know  from  the  experiments. 

Part  I:  Developing  Terminology 

The  Principle:  The  criteria  for  good  terminology  to  use  with  unfamiliar 
objects,  actions,  or  situations  are  that  the  terminology: 

(a)  be  natural,  so  people  with  no  experience  can  use  It; 

(b)  be  short,  so  that  In  a  verbal  communication,  only  a  few  words  of 
description  are  needed; 

(c)  be  well  remembered;  and 

(d)  form  a  classification  system.  That  is,  names  of  objects  should  contain 
generic  terms  and,  when  necessary,  one  or  more  modifiers. 

We  give  here  the  experimental  method  for  deriving  terminology  which  meets 
the  above  criteria.  Part  of  the  method  Is  described  in  detail  in  [l].  It  Is 
extended  and  Improved  here. 

The  method  for  creating  good  names  for  unfamiliar  objects  Is  an  Iterative 
procedure  with  three  steps: 

Step  1.  Names  are  generated  for  each  of  the  objects  by  a  group  of  subjects. 

Step  2.  From  the  names  generated  by  subjects,  the  experimenter  rhooses  a  subset 

of  the  names,  according  to  the  following  criteria:  (1)  the  modal  name  Is 

) 

chosen,  namely.  If  a  particular  name  Is  generated  more  often  than  others,  .it  is 
chosen;  (b)  shorter  names  are  preferred;  and  (c)  the  names  chosen  stay  within 
the  classification  system  provided  by  the  subjects. 

Step  3.  How  good  the  names  are  Is  tested  by  measuring,  first,  how  well  j>eople 
can  match  the  names  with  the  objects  they  describe,  and  second,  how  wel,l  they 
can  recall  the  names,  given  the  objects. 

Steps  2  and  3  can  be  Iterated:  If  a  given  name  Is  poorly  matched  or 
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recalled.  It  can  be  replaced  by  another  generated  name  and  tested  again. 

In  our  experiment,  the  Items  to  be  named  were  the  48  different  pieces  from 
the  Flscher-Technik  50  assembly  kit.  One  such  piece  Is  shown  In  Figure  1.  It 
Is  red  plastic,  with  an  actual  size  of  15  x  15  x  7.5mm  (.6  x  .6  x  .3  In).  We 
show  here  how  It  was  named. 


Insert  Figure  1  about  here 

In  Step  1,  fourteen  people1  named  It  as  follows:  red  H  block,  all  purpose 
joint,  universal  connector,  X-joInt,  H  piece,  universal  connector,  H  joint, 
holder,  universal  frame  connection,  large  block  connector,  flat  grooved 
connector  (female),  red  ,  flat  bracket  with  grooves,  block  2. 

These  names  were  formed  Into  a  graph,  as  shown  In  the  upper  panel  of  Figure 
2.  The  graph  has  nodes  containing  the  different  words. 


Insert  Figure  2  about  here 


It  also  has  directed  links,  from  A  to  B,  for  all  cases  when  two  words,  A  and  B, 
were  given  consecutively  In  a  name,  with  A  preceding  B.  There  are  also  start 
and  end  nodes.  The  number  of  times  a  particular  word  wa<  used  Is  given  in 
parentheses  under  Its  node,  for  all  words  used  twice  or  more. 

One  has  options  In  forming  the  graph.  For  example,  one  can  decide  to  form 
grammatical  categories,  so  that  "block"  can  occur  on  the  graph  as  both  adjective 
and  noun.  (We  did.)  One  can  decide  to  collapse  the  nodes  "grooved"  and  "with" 
and  "grooves"  Into  one  node,  "grooved".  (We  did.) 

From  this  graph,  a  composite  naming  diagram  was  formed,  as  shown  In  the 
lower  panel  of  Figure  2.  It  Is  a  subgraph  consisting  of  all  nodes  with  words 
mentioned  at  least  twice.  (How  many  times  a  word  must  be  mentioned  In  order  for 
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It  to  appear  on  the  composite  naming  diagram  Is  determined  by  the  experimenter, 
depending  on  the  number  of  subjects  run  and  the  variety  of  words.  We  chose 
two.) 

From  the  composite  diagram,  a  name  was  chosen,  using  the  guidelines  of  (a), 
(b),  (c),  and  (d)  above.  Names  suggested  as  candidates  from  the  diagram  were: 
block,  red  H  block,  red  H  joint,  H  block,  H  joint,  flat  grooved  connector,  and 
universal  connector.  These  were  only  suggestions;  the  experimenter  could 
choose  as  a  name  any  shortened  name  (e.g.,  red  block,  grooved  connector)  or  any 
name  formed  from  unlinked  combinations  of  modifiers  and  noun  (e.g.,  universal 
block,  flat  M  joint).  We  chose  the  name  red  H  joint  for  Iteration  1. 

In  a  similar  manner,  a  name  was  selected  for  each  of  the  other  48  pieces. 
These  are  called  Iteration  one  names.  The  48  Iteration  one  names  were  used  to 
begin  the  Iterative  procedure.  That  is,  they  were  tested  (using  new  subjects) 
for  matching  and  recall.  In  scoring  the  matching  and  recall  tasks,  the  errors 
clearly  Indicated  misleading  names.  These  names  were  changed  for  the  next 
Iteration.  Usually  a  new  name  from  the  composite  naming  diagram  was  selected. 
Sometimes,  when  the  composite  naming  diagram  did  not  suggest  a  new  name,  more 
subjects  generated  names  for  the  plece(s),  and  a  new  name  was  chosen  frbm  the 
new  composite  naming  diagram. 

If  a  new  name  Involved  a  change  in  category  for  a  piece  (is  "strip"  to 
"rail"  or  "plate"  to  "platform"),  names  of  all  other  pieces  In  that  category 
were  changed  to  the  new  one. 

The  names  for  the  piece  In  Figure  1  were  red  joint,  grooved  H  joint,  and 

H  ^°1nt  f°r  Iterations  1,  2,  and  3  respectively.  (The  manufacturer's  name  for 
It  Is  building  block  7.5.) 

Percentage  correct  for  the  48  names  on  matching  and  recall,  and  the  average 
number  of  words  per  name,  are  given  In  Table  1  for  each  of  the  three  iterations, 
and  for  the  names  appearing  In  the  manufacturer's  Instruction  booklet.  Table  1 


shows  that  In  general,  as  Iterations  progressed,  names  became  shorter  and  were 
better  watched  with  their  physical  referents  and  better  recalled.  All  groups 
with  subject-derived  names  (Iterations  1,  2,  and  3)  substantially  outperformed 
the  group  with  the  manufacturer's  names. 

The  number  of  Iterations  needed  to  derive  the  names  will  probably  vary  with 
the  Items  to  be  named.  In  our  study,  only  three  Iterations  were  used  because 
the  score  on  the  matching  task  on  Iteration  three  was  nearly  100%  and  therefore 
could  not  be  significantly  Increased. 

This  technique  to  derive  good  names  has  two  nice  properties: 

(1)  It  gets  around  the  problem  of  having  to  specify  what  should  (always)  and 

should  not  (ever)  be  Included  In  a  name.  For  example.  It  does  not  specify  If 

color,  size,  or  shape  should  be  Included. 

(2)  It  Is  subject-driven.  The  names  elected  will  probably  reflect  subjects' 
linguistic  abilities  and  preferences. 

A  feature  of  a  piece  Is  a  part  of  the  piece  which  needs  a  name  In 
Instructions  for  assembly.  Examples  are  knob,  groove,  teeth,  and  slot.  These 
names  were  derived  as  follows. 

The  same  methodology  used  for  the  naming  schema  (but  without  the 

Iterations)  was  used.  That  Is: 

(1)  Subjects  generated  names  for  the  features. 

(2)  New  subjects  were  given  the  feature  names  and  ranked  them  according  to 

their  preference. 

(3)  The  feature  was  given  the  name  which  was  most  preferred. 

Here  Is  an  Important  finding:  In  most  cases,  the  most  frequently  generated 
feature  name  got  the  most  first  place  votes  (or  the  highest  mean  rank  ordering). 
But  In  a  few  cases,  a  less  frequently  generated  name  won.  This  means  that, 
although  people  cannot  necessarily  generate  the  most  preferred  name,  they  can 
nevertheless  recognize  It. 
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To  derive  descriptions  of  actions  required  to  join  pieces,  a  stellar 
methodology  was  used: 

1.  Subjects  learned  the  names  derived  above  for  pieces  and  their  features. 

2.  They  studied  diagrams  and  actual  pieces  in  each  of  two  states,  unassepbled 
and  assembled. 

3.  They  went  through  the  action  with  the  actual  pieces,  from  unassembled  to 
assembled,  five  times. 

4.  They  wrote  down  what  they  did  In  the  form  of  Instructions. 

These  data  showed  that  of  the  three  parts  necessary  for  a  full  description, 
that  Is,  (1)  Initial  condition;  (2)  action;  (3)  final  condition,  about  1/4  of 
the  subjects  described  (1)  and  (2),  leaving  (3)  unspecified,  and  about  3/4  of 
the  subjects  described  (2)  and  (3),  leaving  (1)  unspecified.  We  do  not  know  at 
the  present  time  which  elements  of  the  action  descriptions  will  give  the  best 
learning  results.  We  also  do  not  know  if  the  most  frequently  generated  verbs 
used  to  describe  the  actions  are  the  most  preferred. 

We  have  given  the  methodology  to  derive  names  for  pieces,  feature  names, 
and  action  descriptions  that  ought  to  be  easily  matched  with  their  visual 
counterparts.  This  methodology  has  already  been  successfully  applied  In  other 
situations  {[2]  and  [3])  where  naming  schemas  are  needed,  and  it  ought  to  be 
useful  in  new  situations  as  well. 

The  first  principle,  then,  states  the  criteria  for  a  good  system  of 
terminology.  And  the  methodology  to  derive  such  terminology  is  given. 

fill  ill  The  Corr«ct  Temporal  Overlap  of  Visual  and  Spoken  Elements  In  a 
Presentation 

The  Principle:  In  order  for  good  associations  between  the  visual  and 
spoken  elements  In  a  presentation  to  occur,  the  visual  part  should  precede,  or 
be  In  synchrony  with,  thr  spoken  pat and  not  follow  It. 
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This  general  principle  does  not  require  additional  experiments  for  Its 
Implementation.  It  can  simply  be  used  as  stated. 

We  describe  briefly  the  experiment  we  performed,  from  which  we  derived  the 
principle.  A  full  version  of  the  experiment  Is  given  In  [4].  A  related 
experiment,  using  educational  material.  Is  in  [5]. 

Fourteen  groups  of  subjects  were  shown  a  thirty  minute  film  which 
Introduced  the  Flscher-Technik  50  assembly  kit,  its  pieces,  their  names  (the 
iteration  three  names  derived  above),  and  some  of  their  uses.  The  film's 
visuals  and  narration  could  be  presented  in  synchrony,  or  one  could  be  shifted 
relative  to  the  other  up  to  21  sec. 

Subjects  saw  the  film  in  one  of  seven  versions:  visuals  moved  relative  to 
narration  by  -21,  -14,  -7,  0  (synchrony),  7,  14,  or  21  sec.  They  were  tested 

Immediately  or  after  seven  days  for  recall  of  the  names,  given  the  pieces.  The 
hypothesis  was,  the  higher  the  recall,  the  better  the  associations. 

The  results  are  illustrated  In  Figure  3.  Scores  were  highest  immediately 


Insert  Figure  3  about  here 


and  after  seven  days  for  two  groups:  synchrony  and  visuals  7  sec  before 
narration.  On  the  Immediate  tept,  each  of  the  other  five  groups  scored  about 
80*  of  the  highest  groups.  On  the  test  after  seven  days,  the  other  five  groups 
scored  differently:  the  three  narration-first  groups  performed  about  30*  less 
well  than  the  two  vlsuals-flrst  groups.  (The  statistical  analyses,  and  a 
theoretical  Interpretation  of  the  results,  are  given  In  [4].) 

The  temporal  order  In  which  visual  and  auditory  elements  were  presented 
differentially  Influenced  the  formation  of  visual-verbal  associations.  When 
visuals  precede  narration  by  up  to  7  sec,  recall  Is  as  good  as  when  visuals  and 
narration  are  in  synchrony.  When  narration  precedes  visuals  by  7  sec  or  more. 


Si 


m  v';r 


much  of  the  narration  is  lost,  especially  after  a  delay. 

To  repeat,  then,  the  principle  of  how  to  overlap  visual  and  spoken  material 
in  time.  In  order  for  good  associations  to  be  formed,  is:  The  spoken  material 
should  follow,  or  be  in  synchrony  with,  the  visual  Image,  and  not  precede  it. 
The  correct  temporal  overlap  of  visuals  and  narration  should  not  be  restricted 
only  to  films.  It  should  hold  as  well  for  illustrated  lectures,  slide  shows, 
written  text  with  pictures,  etc.  One  should  present  the  visual  part  early,  or 
simultaneously  with  the  text.  Show  first  and  tell  second,  or  show  and  tell  in 
synchrony,  but  do  not  tell  first  and  show  second. 

Part  1*1:  Dividing  Instructional  Material  into  Conceptual  Units. 

The  Principle:  Decomposition  of  instructional  material  into  conceptual 
units  should  be  in  agreement  with  people's  natural  conceptualization  of  the 
task. 

In  order  to  implement  this  principle,  three  steps  are  required: 

1.  Find  what  the  natural  conceptualization  of  a  person  is. 

2.  Find  if  different  people  conceptual ize  uniformly  (If  they  do  not,  probably 

different  conceptualizations  of  the  material  are  required  for  different  people.) 

3.  Arrange  the  material  to  be  presented  according  to  the  subjects' 

conceptualization. 

Below,  we  present  the  experimental  methods  for  steps  '1)  anq  (2).  Namely, 

we  present  first  the  technique  for  finding  an  individual  subject's 

conceptualization.  We  then  present  the  technique  for  determining  if  subjects 
conceptualize  uniformly,  and  for  constructing  a  composite  conceptualization  for 
a  population  of  subjects.  (Step  two  requires  extensive  programming.) 

Step  1:  Finding  the  natural  conceptualization  of  an  individual. 

We  outline  here  a  methodological  schema  to  find  how  people  divide  an  object 
Into  subassemblies,  that  Is,  how  they  conceptualize  it,  from  the  order  in  which 
they  use  the  parts  in  the  construction  of  the  object.  The  assumption  we  are 
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making  can  be  Illustrated  by  a  simple  example.  If,  In  joining  four  pieces.  A, 
B,  C,  and  D,  a  person  consistently  joins  A  and  B,  and  then  C  and  D,  and  then 
joins  the  two  subassemblies.  It  Is  expected  that  In  a  division  Into  two  parts, 
the  person  has  the  concepts  (AB)  and  (CO). 

The  method  used  Is  to  have  a  person  ask  for  pieces  one  at  a  time  for 
assembly,  and  to  record  the  order  of  request.  It  has  the  following  underlying 
hypothesis:  In  assembling  an  object  from  a  model  or  other  Input,  the  person 
conceptualizes  the  object  to  be  built,  and  then  asks  for  parts,  grouped  together 
according  to  the  conceptual  division. 

These  data  are  easy  to  gather,  even  for  complex  objects.  We  will  show  data 
from  an  object  (the  toy  helicopter  shown  In  Figure  4)  consisting  of  54  pieces, 
but  we  estimate  that  substantially  more  pieces  do  not  create 


Insert  Figure  4  about  here 

a  problem.  The  data  analysis  is  also  straightforward.  It  consists  of  three 
parts: 

1.  An  assembly  object  Is  drawn  as  an  abstract  graph  whose  nodes  represent 
pieces  and  whose  edges  (links)  represent  connections.  (This  representation  can 
be  used  on  any  assembled  object,  not  just  Fischer  Technik.l  The  abstract  graph 
of  the  helicopter  shown  In  Figure  4  is  given  In  Figure  S.L  Nodes  In  Figure  5 
are  numbered  1  through  54,  to  correspond  to  specific  pieces  in  the  helicopter. 


Insert  Figure  5  about  here 


2.  A  distance  between  nodes  on  the  graph  is  Introduced,  based  on  how  closely 
the  requests  for  the  different  pieces  are.  (For  example.  If  a  person  requests 
piece  10  fifth  and  piece  11  ninth,  the  distance  between  pieces  10  and  11  Is  |  5  - 


3.  A  cluster  analysis  Is  performed,  and  the  clusters  are  used  as  hypothetical 
conceptual  units  of  the  person  building.  Each  node  is  put  In  a  cluster  with  its 
closest  connected  neighbor.  An  example  is  given  In  Figure  6  by  the  thin  solid 
lines  on  the  figure.  Then  each  cluster  Is  put  in  a 


Insert  Figure  6  about  here 


higher-order  cluster  with  Its  closest  connected  neighbor.  These  are  the  dotted 
lines  on  the  figure.  Each  of  these  is  put  in  an  even  higher-order  cluster  (the 
heavy  solid  lines  on  the  figure).  The  process  is  continued  until  all  clusters 
fall  into  the  same  higher  order  cluster. 

This  analysis  yields  a  hierarchical  tree,  which  is  the  hypothetical  natural 
conceptualization  of  the  object  by  an  individual. 

Step  2:  Finding  if  different  people  conceptualize  uniformly. 

Below  we  give  a  method  to  determine  how  different  conceptualizations  from 
different  people,  and  from  one  person  on  different  trials,  are.  That  is,  are 
they  minor  variants  of  the  same  conceptualization,  or  do  they  form  different 
categories?  He  demonstrate  the  method  in  the  context  of  the  experiment  we 
conducted. 

Sixteen  people  built  the  helicopter  five  times,  once  every  other  day.  A 
physical  model  was  used  as  a  guide  on  each  trial.  Each  time,  the  subject  was 
required  to  request  each  piece  separately,  and  the  order  of  request  was 
recorded.  A  person's  conceptualization  of  the  helicopter  was  derived  from  the 
order  of  requests,  as  described  above,  using  a  computer  package  ([6]). 

Among  the  80  trials  (16  subjects  x  5  trials  each),  all  conceptualizations 
were  different.  The  questions  we  were  able  to  answer  were: 

1)  Can  different  conceptualizations  be  treated  as  variants  of  one 
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conceptualization,  or  do  they  form  different  categories? 

2)  How  does  the  conceptualization  presented  In  an  Instructional  film  we  are 
using  compare  with  subjects'  conceptualizations? 

The  method  used  was  a  cluster  analysis  of  the  81  trials,  including  the 
conceptualization  from  the  film.  The  distance  between  trials  Is  described  In 
the  Appendix. 

The  main  result  Is  that  the  population  of  trials  divided  Into  one  large 
cluster  of  66  cases,  and  three  others,  having  11,  2,  and  2  cases  respectively. 
The  conceptualization  presented  in  the  instructional  film  went  into  the  largest 
cluster. 

For  a  composite  graph,  the  average  distance  between  nodes  is  computed.  The 
composite  conceptualization  from  the  66  cases  is  shown  in  Figure  6. 

Our  major  finding  is  that  over  80%  of  the  trials  (66  of  81)  fall  into  the 
same  cluster.  This  finding  is  important  for  individualized  instruction.  When  a 
collection  of  trials  splits  into  many  different  clusters,  it  means  that 
different  people  conceptual 1 ze  differently,  and  that  one  person  conceptualizes 
differently  at  different  times.  That  indicates  that  in  order  to  improve 
performance,  instructions  need  to  be  tailored  specifically  for  a  person  in  a 
given  situation.  The  fact  that  80%  of  the  trials  fall  in  one  cluster  indicates 
that,  at  least  for  the  subject  population  tested  and  the  object  *u11t  here,  one 
set  of  Instructions  can  cover  a  majority  of  people.  (We  have  obtained  a  similar 
result  using  a  different,  more  complex,  object  in  another  study.  There,  the 
majority  cluster  contains  70%  of  the  trials.) 

The  fact  that  the  conceptualization  from  our  film  (used  in  Part  IV)  falls 
Into  the  largest  cluster  means  that  it  follows  Principle  3.  Its 
conceptualization  Is  the  same  as  that  of  the  majority  of  the  people  who  will  be 
Instructed  by  It. 

In  Part  III  we  have  given  the  principle  (to  be  tested  in  future  work)  that 
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the  conceptual  units  given  In  Instructions  should  conform  to  people's  natural 
conceptualization.  And  we  have  given  the  methodology  to  find  If  people 
conceptualize  uniformly,  and  the  technique  for  constructing  a  composite 
conceptualization  for  a  group  of  subjects. 

Part  IV;  Learnl ng  *  Procedure  from  Multimedia  Instructions;  The  Effects  of 
Filii  and  Practice. 

The  Principle.  For  good  retention  of  a  procedure  to  be  performed  from 
memory,  the  arrangement  of  an  Instructional  sequence  consisting  of  film  and 
practice  should  be  practice  first  and  film  second.  This  is  a  rule  of  thumb,  to 
be  used  when  no  Information  Is  known  about  the  person  being  trained.  When 
variables  such  as  manual  dexterity  and  experience  with  similar  tasks  have  been 
assessed,  a  training  sequence  differing  from  practice  first,  film  second  may  be 
better  for  a  particular  Individual. 

We  present  here  a  summary  of  the  experiment  on  which  we  base  the  principle. 
The  details  are  In  [7].  A  related  study,  using  only  pictures  and  text  for 
Instructions,  Is  In  [8]. 

Different  modalities  of  Instruction  (film  versus  practice),  different 
amounts  of  the  two,  and  different  orders  (film  first  or  practice  first)  were 
given  to  people  In  the  experiment.  By  practice  we  mean  that  people  built  the 
object  with  a  physical  model  sitting  before  them  as  a  gt^de.  The  object  to  be 
assembled  was  the  54  piece  helicopter  shown  In  Figure  4.  The  12  groups,  their 
Instructional  sequences,  and  their  time  of  test,  are  given  in  Table  2. 


Insert  Table  2  about  here 


The  Instructional  film,  shot  by  James  Otis,  was  15  min  long.  In  color,  and 
narrated.  The  conceptual  units  presented  In  the  film  were  the  same  as  those  of 


the  majority  of  the  people  who  built  the  helicopter  from  a  model.  In  the  work 


..i  \f" 


“  'is  i&t 


presented  In  Part  III. 

After  the  Instructions,  Including  practice  where  appropriate,  each  person 
was  required  to  build  the  helicopter  from  memory,  either  Immediately  or  after  a 
one  week  delay.  Note  that  the  four  groups  Instructed  by  film  alone  did  not  have 
hands-on  practice  during  training.  Th^y  built  the  helicopter  only  once,  from 
memory.  All  other  groups  built  the  helicopter  at  least  once  during  training, 
using  a  model  as  a  guide.  They  built  It  again,  this  time  from  memory,  during 
the  test  trial . 

Performance  on  the  memory  trial  was  assessed  as  follows:  The  abstract 
graph  of  each  helicopter  built  from  memory  was  drawn.  The  number  of  correct 
connections  It  contained  was  the  dependent  measure.  (This  assesses  the 
similarity  In  structure  of  the  helicopter  built  from  memory  and  the  correctly 
built  helicopter.)  There  are  58  connections  in  the  correctly  built  helicopter 
(as  can  be  seen  In  Figure  5),  so  the  range  was  0  to  58. 

The  results  are  given  In  Table  3.  For  convenience  In  talking  about  the 
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groups,  we  abbreviate  film  by  F  and  model  by  M.  For  example,  the  groups  who, 
during  training,  saw  the  film  first  and  built  the  model  second,  »re  abbreviated 
FM. 

A  Newman-Keuls  procedure  was  used  to  test  differences  between  pairs  of 
means  at  zero  delay.  (See  [9].)  A  separate  procedure  was  used  for  7-day  delay. 
The  groups  who  built  the  helicopter  Immediately  after  their  Instruction  line  up 
statistically  as  follows  with  respect  to  their  performance  from  memory: 
MM«MF»FN>FF-M>F. 

This  result  means  that  some  practice  Is  good  during  Instruction,  either  building 
twice  or  building  once  and  seeing  a  film.  (Order  of  practice  and  film  does  not 


Mtter  when  performance  is  tested  iMwdietely.) 

After  •  seven  day  delay,  the  lineup  of  the  groups  Is  different: 

Mf>MN*FMsM*FF>F. 

All  groups  are  depressed  to  about  50%  of  their  scores  when  tested  Immediately, 
except  for  one,  the  group  that  builds  first  and  sees  the  film  second.  Its 
performance  after  a  week  Is  30.3/46.7  *  65%  of  Its  performance  at  zero  delay. 
Retention  of  a  procedure  to  be  performed  from  memory  is  clearly  highest  in  this 
group.  In  general,  when  a  person  builds  first  and  then  sees  a  film  displaying 

3 

conceptual  units,  with  names,  second,  his  or  her  performance  Is  best. 

However,  Individual  differences  in  performance  within  a  group  were  very 
great.  For  example,  scores  could  range  from  0  to  58,  and  an  actual  range  in  a 
single  group  of  2  to  56  was  common.  The  average  standard  deviation  In  a  group 
was  over  20%. 

This  finding  leads  us  to  conclude  that  the  right  training  sequence  for  a 
procedure  that  Is  to  be  performed  from  memory  varies,  depending  on  the 
Individual.  And  this  brings  up  the  question  of  Individualized  Instruction.  A 
goal  of  our  future  research  Is  to  discover  what  Individualized  Instruction 
should  contain.  Specifically,  should  Instruction  be  individual ized  simply  by 
varying  the  amount  given  to  different  people,  depending  on  their  experience  or 
skill?  Or  should  It  be  individualized  by  giving  different  "localities,  or 
modalities  In  different  orders,  or  different  conceptualizations,  etc.?  A  second 
goal  of  our  future  work  Is  to  develop  a  small  number  of  brief  tests  which  can  be 
easily  given  to  subjects.  Performance  on  these  tests  would  be  used  to  (a) 
predict  performance  as  a  function  of  Instructions;  and  (b)  assign  a  person  to 
an  appropriate  Instructional  sequence. 

Until  such  tests  are  available,  we  recommend  that  a  person's  performance  be 
tested  after  practice,  after  film  Instruction,  and  after  various  amounts  and 
combinations,  to  see  which  gives  optimum  results.  If  such  testing  Is  not 


possible,  the  Instructional  sequence  should  be  practice  first  and  fit*  second. 

Final  Renarks 

The  four  principles  presented  In  this  paper  were  derived  from  and  tested  on 
prlnarlly  assembly  tasks.  Their  general liability  to  other  types  of  tasks,  for 
exanple,  repair  tasks,  programing,  use  of  new  equipment,  etc.,  should  be  tested 
experimentally.  .The  methodologies  given  here  can  be  easily  modified  for 
studying  the  tasks  mentioned  above.. 
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Footnotes 

This  research  was  supported  by  the  Office  of  Maval  Research  Contract 
#W00014-78-C-0433,  MR  157-422.  This  paper  Is  Technical  Report  #121  of  the 
Institute  of  Cognitive  Science,  University  of  Colorado  at  Boulder. 

1.  In  this  and  all  other  experiments  reported,  the  subjects  were  students 
enrolled  In  Introductory  psychology  at  the  University  of  Colorado  who 
participated  a?  part  of  a  class  requirement. 

2.  The  connections  to  be  considered  can  be  set  for  each  analysis.  Here  we 
consider  only  physical  connections.  There  are  58  In  the  helicopter.  We  could 
have  considered  as  many  as  ( ■  1431. 

3.  In  our  experiment,  we  put  a  limit  on  the  type  and  amount  of  Instruction. 
The  theoretical  rationale  for  this  Is  given  In  [7].  When  there  Is  no  such 
limit,  longer  sequences,  such  as  practice  first,  film  second,  practice  third, 
might  prove  even  better  than  the  arrangement  suggested  here. 
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Figure  Captions 

Figure  1.  A  piece  from  the  assembly  kit.  Its  actual  size  Is  15  x  15  x  7.5am 
(.6  x  .6  x  .31n). 

Figure  2.  Upper  panel:  A  graph  of  the  14  names  generated  for  the  piece  shown 
In  Figure  1.  The  nodes  contain  the  different  words.  The  links  are 
directed  from  A  to  B,  for  all  cases  when  two  words,  A  and  B,  were  given 
consecutively  In  a  name,  with  A  preceding  B.  The  number  of  time  a 

particular  word  was  used  Is  given  In  parentheses  under  Its  node,  for  all 
words  used  twice  or  more. 

Lower  panel:  A  composite  naming  diagram.  It  Is  a  subgraph  consisting 
of  all  nodes  with  words  mentioned  at  least  twice.  Names  for  the  piece  In 
Figure  1  suggested  as  candidates  from  the  diagram  are:  block,  red  H  block, 
red  H  Joint,  H  block,  H  joint,  flat  grooved  connector,  and  universal 
connector. 

Figure  3.  Percentage  correct  on  recall  of  names,  given  the  pieces,  as  a 
function  of  degree  of  asynchrony  between  the  visual  and  spoken  material  in 
the  film,  and  delay  between  the  film  and  the  test  (zero-  or  7-day). 

Figure  4.  A  toy  helicopter  built  from  54  pieces  of  the  Flscher-Technlk  50 
assembly  kit. 

Figure  5.  An  abstract  graph  of  the  toy  helicopter  shown  In  Flour'*  4.  The  nodes 
represent  pieces  In  the  helicopter,  and  the  links  represent  physical 
connections. 

Figure  6.  The  composite  conceptualization  of  the  helicopter  from  the  majority 
group  (66  of  81  trials).  The  method  for  obtaining  this  division  Into 

conceptual  units  is  given  In  the  text. 

Figure  7.  Pieces  p}  and  p2  occur  In  conceptualizations  T^  and  Tj  as  show*.  In 
Tj,  pj  and  pg  are  In  the  same  first  order  cluster,  so  that  their  height 
equals  one.  In  Tj,  they  are  in  the  same  second  order  cluster,  so  that 
their  height  equals  two. 
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Appendix 

There  are  two  steps  In  doing  the  cluster  analysis  on  a  group  of 
conceptualizations.  Both  are  done  using  the  computer  package  In  [6]. 

1.  Find  the  distance  between  all  pairs  of  conceptualizations; 

2.  Do  a  cluster  analysis  on  the  space  of  all  pairs  of  conceptualizations,  with 
distances  defined  from  step  1. 

The  details  required  for  each  step  are  given  below: 

1.  The  distance  between  conceptualization  on  two  trials  T.  and  Tj  Is  defined  as 
follows: 

It  Is  the  sum  (over  all  58  connected  pairs  of  pieces  in  the  helicopter)  of  the 
difference  In  height  In  a  conceptualization  necessary  to  put  a  connected  pair  In 
the  same  cluster. 

Here  Is  an  example.  Consider  a  pair  of  connected  pieces  Pj  and  p,,. 
Suppose  they  are  placed  In  the  conceptualizations  of  T.  and  T.  as  shown  In 
Figure  7.  In  conceptualization  Tj,  pj  and  pg  are  In  the  same  first  order 
cluster.  Their  height  *  1.  In  conceptualization  Tj,  Pj  and  p2  are  In  the 

Insert  Figure  7  about  here 


same  second  order  cluster  (dotted).  Their  height  •  2. 

The  distance  between  the  pair  of  pieces  (pj.Pg)  In  conceptualizations  T^  and  T^ 

Is  the  difference  In  their  heights,  2-1  -1. 

The  distance  between  Tj  and  Tj  Is  the  sum  (over  all  58  pairs)  of  these 
distances. 

i 

j 

j 


2.  A  cluster  analysis  Is  dona  on  the  conceptualizations,  with  each  one  put  In 
cluster  with  Its  closest  connected  neighbor  !as  described  In  Part  3). 
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Table  1:  Percentage  Correct  on  Matching  and  Recall,  and  Average 
Humber  of  Words  per  Name,  for  Each  of  Four  Groups 

percentage  percentage  average 

correct:  correct:  number  of 

matching  surprise  words  per  name 

recall* 

group  given  59.89  27.25  2.94 

names  from 

manufacturer 

group  given  89.20  48.64  2.75 

Iteration  1 

names 

group  given  93.92  48.60  2.81 

Iteration  2 

names 

group  given  96.23  50.72  2.60 

Iteration  3 

names 

♦No  variation  was  scored  as  correct.  For  example,  for  the  triangle  joint,  the 
name  triangular  Joint  was  scored  as  wrong. 


Table  2:  Experimental  Groups  for  Mixing  Modalities  In  Instruction 


stimulus  1 

see 

build 

see 

build  . 

film 

from 

film 

from 

model 

model 

stimulus  2  build 

see 

see 

build 

see 

build 

from 

film 

film 

from 

film 

from 

model 

again 

model 

model 

again 

test  (Immediately,  for  6  groups)  build  helicopter  from  memory 

test  (after  1  week,  for  6  groups)  build  helicopter  from  memory 


Table  3:  Mean  Number  of  Correct  Connections  in  Helicopter  Built 
From  Memory  {a  score  of  58  is  possible) 


Figure  1 


narration 
*1  aac 
bafora 
v law  ala 


narration 
14  aac 
bafora 
vlauala 


narration  narration  vlauala 

7  aac  and  vlauala  7  aac 
bafora  in  synchrony  bafora 
vlauala  narration 


vlauala 
>4  sac 
bafora 
narration 


vlauala 
tt  aac 
bafora 

narration 


TEMPORAL  SHIFT 


Figure  3 
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Figure  6 


In  conceptualization  1]: 


Figure  7 
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Operations  1  raining  Division 
WILLIAMS  AFB,  AZ  85224 

1  Dr.  Stanley  Col Iyer 
Office  of  Naval  Technology 
800  N.  Quincy  Street 
Arlington,  VA  22217 

1  CDR  Mike  Curran 
Office  of  Naval  Research 
800  N.  Quincy  St. 

Code  270 

Arlington.  VA  22217 

1  Dr.  Tom  Duffy 

Navy  Personnel  RAD  Center 
San  Diego.  CA  92152 

1  Edward  E.  Eddowes 

Education.:!  Adviser,  Code  N-3 
HQ  Chief  of  Naval  Air  Training 
Corpus  Christ i,  TX  78419 

1  DR.  PAT  FEDERICO 
Code  PI 3 
NPRDC 

San  Diego.  CA  9215? 

1  Dr,  John  Ford 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Dr.  Mike  Cayr.or 

Navy  Research  Laboratory 
Code  7510 

Washington,  DC  20375 

1  Dr.  Jirr  Hoi lan 
Code  304 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Dr.  Ed  Hutchins 
Navy  Personnel  RAD  Center 
Ssn  Diego,  CA  9215? 

1  Dr.  Norman  J.  Kerr 
Chief  of  Naval  Technical  Training 
Naval  Air  Station  Memphis  (75) 
Millington,  TN  ’8054 
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Navy 

.1  Dr.  feter  Kincaid 

Training  Analysis  A  Evaluation  Group 
Dept,  of  the  Navy 
Orlando,  FL  j281j 

1  Dr.  Ray  Main 
Code  14 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Dr.  Willlem  L.  Maloy 

Principal  Civilian  Advisor  for 
Education  and  Training 
Naval  Training  Command,  Code  00A 
Pensacola,  FL  3250B 

1  CAPT  Richard  L.  Martin,  USN 
Commanding  Officer 
ICS  Carl  Vinson  (CVN-70) 

FRO  Mew  York,  NY  09558 

1  Dr  William  Montague 
NPRDC  Code  13 
San  Diego.  CA  92152 

1  Bill  Nordbrock 
1032  Fairlawn  Ave. 

Libertyville,  IL  80048 

1  Library.  Code  P201L 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Technical  Director 
8avy  Personnel  RAD  Center 
&• n  Diego,  CA  9215? 

6  Commanding  Officer 

riaval  Research  Laboratory 
iodp  2627 

Washington,  DC  20390 

1  Office  of  Naval  Research 
Code  «33 

800  N.  Quincy  SStreet 
Arlington,  VA  22217 

6  Personnel  A  Training  Research  Group 
Code  442PT 

Office  of  Naval  Research 
Arlington,  VA  22217 


Navy 

1  Psychologist 
ONR  Branch  Office 
1030  East  Green  Street. 

Pasarteru  .  CA  91101 

1  Special  Asst,  for  Education  and 
Training  (0P-01E) 
ta.  2705  Arlington  Annei 
Washington,  DC  20370 

1  Office  of  the  Chief  of  Naval  Operations 
Research  De/elopment  A  Studies  Branch 
OP  115 

Washington,  DC  20350 

1  Dr.  Gary  Poock 

Operations  Research  Department 
Code  55PK 

Naval  Postgraduate  School 
Monterey,  CA  93940 

1  Dr.  Gil  Rioard 
Code  N71 1 
NTEC 

Orlando,  FL  32613 

1  Dr.  Carl  Ross 
CNET-PDCD 
Building  90 

Great  Lakes  NTC,  IL  60038 

1  Dr.  Worth  Scan  land 
CNET  (M-5) 

NAS,  Pensacola,  FL  32503 

1  Mr.  Irving  Schiff 
Dept,  of  the  Navy 
Chief  of  Naval  Operations 

OP  113 

Washington,  DC  2C350 

1  Dr.  Robert  G.  Smith 

Office  of  Chief  of  Naval  Operations 
OP-987H 

Washington,  DC  ?C;;50 

1  Dr.  Alfred  F.  Smode,  Director 

Training  Analysis  A  Evaluation  Group 
Dept,  of  the  Navy 
Orlando,  FL  3281 3 

1  Dr .  Richard  Sorensen 
Navy  Personnel  RAD  Center 
San  Diego,  CA  92152  j 
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Navy 

1  Dr.  Frederick  Steinheiser 
CNO  -  OP115 
Navy  Annex 
Arlington.  VA  20370 

1  Code  14 

Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 

1  Roger  Weissinger-Baylon 
Department  of  Administrative  Sciences 
Naval  Postgraduate  School 
Monterey,  CA  93940 

1  Mr  John  H.  Wolfe 
Navy  Personnel  RAD  Center 
San  Diego,  CA  92152 


Marine  Corps 

1  H.  Will  ism  Greenup 

Educ.it ion  Advisor  (E0J1) 
Education  Center,  KCDEC 
Quant ioo,  VA  22134 

1  Special  Assistant  for  Marine 
Corps  Matters 
Code  10GM 

Office  of  Naval  Research 
800  N.  Quincy  St. 

Arlington,  VA  22217 

1  DR.  A.L.  SLAFK0SKY 
SCIENTIFIC  ADVISOR  (CODE  RD-1 ) 
HQ,  U.S.  MARINE  CORPS 
WASHINGTON,  DC  20380 
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Amy 

1  Technical  Director 

U.  S.  Amy  Research  Institute  for  the 
Behavioral  8rtd  Social  Sciences 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Mr.  James  Baker 

Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Dr.  Beatrice  J.  Farr 
U.  S.  Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22 333 

1  Dr.  Milton  S.  Katz 
Training  Technical  Area 
U.S.  Array  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  br.  Harold  F.  O'Neil,  Jr. 

Director,  Training  Research  Lab 
Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

1  Joseph  Psotka.  Ph.D. 

TTM:  PERI-1C 
rmy  Research  Institute 
5001  Eisenhower  Ave. 

Alexandria,  VA  22333 

1  Or.  Robert  Sasmor 
U.  S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 
50C1  Eisenhower  Avenue 
Alexandria,  VA  2233? 

1  Dr.  Robert  Wisher 

Army  Research  Institute 
5001  Eisenhower  Avenue 
Alexandria,  VA  ?2’33 


Air  Force 

1  AFHRL/LRS 

Attn:  Susan  Ewing 
WPAFB 

WPAFB,  OH  454^} 

1  U.S.  Air  Force  Office  of  Scientific 
Research 

Life  Sciences  Directorate,  NL 
Bolling  Air  Force  Base 
Washington,  DC  2033 2 

1  Air  University  Library 
AUL/LSE  76/443 
Maxwell  AFB,  AL  36112 

1  Dr.  Earl  A.  Alluisi 
HQ,  AFHRL  (AFSC) 

Brooks  AFB,  TX  78235 

1  Bryan  Da  liman 
AFHRL/LRT 

Lowry  AFB,  On  302  "0 

1  Dr.  Alfred  R.  Fr< gly 
AFOSR/NL 

Bolling  AFB,  DC  ?0?;.2 

1  Dr.  Genevieve  Haddad 

Program  Manager 

Life  Sciences  Directorate 

AFOSR 

Bolling  AFB,  DC  20T'2 

1  Dr.  T.  M.  Longridge 
AFHRL/OTGT 

Williams  AFB.  AZ  P.5224 

1  Dr.  Joseph  Yasatuke 
AFHRL/LRT 

Lowry  AFB.  CO  30230 
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Department  of  Defense 

12  Defense  Technical  Information  Center 
Cameron  Station,  Bldg  5 
Alexandria,  VA  22’  14 
Attn:  TC 

1  Military  Assistant  for  Training  and 
Personnel  Technology 
Office  of  the  Under  Secretary  of  Defen 
for  Research  4  Engineering 
Room  3D  129,  The  Pentagon 
Washington,  DC  20301 

1  Major  Jack  Thorpe 
DARPA 

1400  Wilson  Blvd. 

Arlington,  VA  22209 


Civilian  Agencies 

1  Dr.  Patricia  A.  Eutler 
NIE-BRN  Bldg,  Stop  8  7 
1200  19th  . .  NW 
Washington,  PC  202 OS 

1  Dr.  Susan  Chipmun 

Learning  and  Development 
National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20203 

1  Edward  Fsty 

Department  of  Education,  0ERI 
MS  40 

1200  19th  St.,  NW 
Washington,  DC  20208 

1  Dr.  John  Mays 

National  Institute  of  Education 
1200  19th  Street  NW 
Washington,  DC  20203 

1  Dr.  Arthur  Melmed 
0ERI 

1200  19th  Street  NW 
Washington,  DC  20203 

1  Dr.  Andrew  R.  Molnar 

Office  of  Scientific  and  Engineering 
Personnel  and  Education 
National  Science  Foundation 
Washington,  DC  20550 

1  Dr.  Judith  Orasanu 

National  Institute  of  Education 
1200  15th  St.,  N.W. 

Washington,  DC  202C3 

1  Dr.  Ramsay  W.  Selden 

National  Institute  of  Education 
120C  19th  St.,  NW 
Washington,  DC  2C203 

1  Chief,  Psychological  Reserch  Branch 
U.  S.  Coast  Guard  (G-P-1 /2/TP42) 
Washington,  DC  20593 

1  Dr.  Frank  Withrow 
U.  S.  Office  of  Education 
400  Maryland  Ave.  SW 
Washington,  DC  20202 
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1  Dr.  Joseph  L.  Young,  Director 
Memory  A  Cognitive  Proces  as 
National  Science  Foundation 
Washington,  DC  20550 


1  Dr.  Erling  R.  Andersen 
Department  of  Statistics 
Stud iestraede  h 
1455  Copenhagen 
DENMARK 

1  Dr.  John  R.  Anderson 
Department  of  Psychology 
Carriegie-Mel Ion  University 
Pittsburgh,  PA  15213 

i  Dr.  John  Annett- 

Department  of  Psychology 
University  of  Warwick 
Coventry  CV4  7AJ 
ENGLAND 

1  Dr.  Michael  Atwood 
Bell  Laboratories 
11900  North  Pecos  St. 

Denver,  CO  80234 

1  Psychological  Research  Unit 
Dept,  of  Defense  ( A'-my  Office) 
Campbell  Park  Offices 
Canberra  ACT  2600 
AUSTRALIA 

1  Dr.  Alan  Badutley 

Medical  Research  Council 
Applied  Psychology  Unit 
15  Chaucer  ko;.c 
Cambridge  CB?  2EF 
ENGLAND 

1  Ms.  Carole  A.  Ragley 

Minnesota  Educational  Computing 
Consortium 

2 354  Hidden  Valley  Lane 
Stillwater,  UN  550^2 

1  Mr.  Avron  B.rr 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  9*»305 

1  Dr.  Gvorge  P.  Pieger 
B-1 10  Coleman  Hall 
Buckn«ll  University 
Lewisburg ,  PA  170'" 
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1  Dr.  John  Black 
Yale  University 

Box  1 1 A ,  Yale  Station 
New  Haven.  CT  06520 

1  Dr.  John  S.  Brown 

XEROX  Palo  Alto  Research  Center 
3333  Coyote  Road 
Palo  Alto,  CA  94304 

1  Bundministerium  der  Verteid igung 
-Referat  P  II  4- 
Psychological  Service 
Postfach  1328 
D-5300  Bonn  1 
F.  R.  of  Germany 

1  Dr.  C.  Victor  Bunderson 
WICAT  Inc. 

University  Plaza,  Suite  10 
1160  So.  State  St. 

Orem,  UT  84057 

1  Dr.  Jaime  Carbonell 

Carnegie-Mellon  University 
Department  of  Psychology 
Pittsburgh,  PA  15213 

1  Dr.  Pat  Carpenter 

Department  of  Psychology 
Carnegie-Mellon  University 
Pittsburgh.  PA  152n 

1  Dr.  William  Chase 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  1521;. 

1  Dr.  Michel ine  Chi 
Learning  R  4  D  Center 
University  of  Pittsburgh 
3939  O’Hara  Street 
Pittsburgh,  PA  15213 

1  Dr.  William  Clancey 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  9a306 


1  Dr.  Michael  Cole 

University  of  California 
at  San  Diego 

Laboratory  of  Comparative 
Human  Cognition  -  D0Q3A 
La  Jolla.  CA  92093 

1  Dr.  Allan  M.  Collins 

Bolt  Beranek  4  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  021 38 

1  Dr.  Kenneth  B.  Cross 
Anacapa  Sciences,  Inc. 

P.0.  Drawer  Q 

Santa  Barbara,  CA  93102 

1  ERIC  Facility-Acquisitions 
4833  Rugby  Avenue 
Be the sd a  ,  MD  20014 

1  Dr.  Paul  Feltovirh 

Department  of  Medical  Education 
Southern  Illinois  University 
School  of  Medicine 
P.O.  Box  3926 
Springfield,  IL  62708 

1  Professor  Reuven  Feuerstein 
HWCRI  Rehov  Karmon  6 
Bet  Hakerem 
JeruSolem- 
Israel 

1  Mr.  Wallace  Feurzeig 

Department  of  Educational  Technology 
Bolt  Feranek  4  Newman 
10  Moul ton  St . 

Cambridge,  HA  0^238 

1  Dr.  Victor  Fields 
Dept,  of  Psychology 
Montgomery  College 
Rockville,  MD  20850 

1  Dr.  Dexter  Fletcher 
WICAT  Research  Institute 
1875  S.  State  St. 

Orem,  UT  223 '•*? 

1  Dr.  John  D.  Folley,  Jr. 

Applied  Science  Associates,  Inc. 

P.  0.  Pox  158 
Valencia,  PA  16059 
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1  Dr.  John  R.  Frcdcriksen 
Bolt  Beranek  A  Newman 
50  Moulton  Street 
Cambridge,  MA  02178 

1  Dr.  Michael  Genesereth 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  94 305 

1  Dr.  Dedre  Centner 
Bolt  Beranek  &  Newman 
10  Moulton  St. 

Cambridge,  MA  02138 

1  Dr.  Robert  Glaser 

Learning  Research  A  Development  Center 
University  of  Pittsburgh 
3939  O’Hara  Street 
PITTSBURGH,  PA  15260 

1  Dr.  Marvin  D.  Clock 
217  Stone  Hall 
Cornell  University 
Ithaca.  NY  14853 

1  Dr .  Josph  Goguen 
SRI  International 
333  Ravenswood  Avenue 
Menlo  Park.  CA  94025 

1  Dr.  Bert  Green 

Johns  Hopkins  University 
Department  of  Psychology 
Charles  A  34th  Street 
Baltimore,  MD  21218 

1  DR.  JAMES  G.  GREEN0 
LRDC 

UNIVERSITY  OF  PITTSBURGH 
3939  O’HARA  STREET 
PITTSBURGH,  PA  15213 

1  Dr.  Barbara  Hayes-Roth 

Department  of  Computer  Science 
Stanford  University 
Stanford,  CA  95305 

1  Dr.  Frederick  Hayes-Roth 
Teknowledge 
525  University  Ave. 

Palo  Alto,  CA  94 301 


1  Dr.  Lloyd  Humphreys 

Department  of  Psychology 
University  of  Illinois 
Champaign,  IL  61820 

1  Dr.  Earl  Hunt 

Dept,  of  Psychology 
University  of  Washington 
Seattle ,  W A  9=105 

1  Mr.  R.  P.  Joyce 

Applied  Science  Associates 
P.  O.  Box  15b 
Valenc  i.', ,  PA  16050 

1  Dr.  Steven  W.  Keele 
Dept,  of  Psyoitology 
University  of  Oregon 
Eugene,  OR  U7401 

1  Dr.  David  <ieras 

Department  of  Psychology 
University  of  Ari7ona 
Tuscon,  AZ  85721 

1  Dr.  Waiter  Kintsch 

Deoartment  of  Psychology 
University  of  Colorado 
Bouldt r ,  CO  80 302 

1  Dr.  Stephen  Kosslyn 

Department  of  Psychology 
Brandeis  University 
Waltham,  MA  02254 

1  Dr.  Pat  Langley 

Carnegie-Mel Ion  University 
Pittsburgh,  PA  15213 

1  Dr.  Marry  L^nsman 
The  L.  L.  Thurstone  Psychometric 
Laboratory 

University  of  North  Carolina 
Davie  Hali  013A 
Chapel  Hill,  NC  27514 

1  Dr.  Jill  Larkin 

Department  of  Psychology 
Carnegie  Mellon  University 
Pittsburgh,  PA  1521?. 
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1  Dr.  Alan  Lesgold 
Learning  RAD  Center 
University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh,  PA  15260 

1  Dr.  Jim  Levin 
University  of  California 
at  San  Diego 

Laboratory  fof  Comparative 
Human  Cognition  -  D003A 
La  Jolla,  CA  92093 

1  Dr.  Michael  Levine 

Department  of  Educational  Psychology 
210  Education  Bldg. 

University  of  Illinois 
Champaign,  IL  M801 

1  Dr.  Marcia  C.  Linn 
University  of  California 
Director,  Adolescent  Reasoning  Project 
Berkeley,  CA  9*720 

1  Dr.  Robert  Linn 
College  of  Education 
University  of  Illinois 
Urbana,  IL  61801 

1  Dr.  Erik  McWilliams 
13216  Ridge  Drive 
Rockville,  MD 
Washington,  DC  20550 

1  Dr.  James  R.  Miller 
Texas  Instruments,  Inc. 

Central  Research  Laboratory 
P.  0.  Box  226015,  MS238 
Dallas,  TX  75266 

1  Dr.  Mark  Miller 

Computer  Thought  Corporation 
1721  West  Plane  Parkway 
Plano.  TX  75075 

1  Dr.  Tom  Moran 
Xerox  PARC 

3333  Coyote  Hill  Road 
Palo  Alto,  CA  9*30* 

1  Dr.  Allen  Munro 

Behavioral  Technology  Laboratories 
18*5  Elena  Ave.,  Fourth  Floor 
Redondo  Beach,  CA  90277 


1  Dr .  Donald  A  Norman 
Cognitive  Science,  C-015 
Univ.  of  California,  San  Diego 
La  Jolla,  CA  92093 

1  Dr.  Jesse  Orlansky 

Institute  for  Defense  Analyses 
1801  N.  Beauregard  St. 

Alexandria,  VA  22311 

1  Dr.  Seymour  A.  Papert 
Massachusetts  Institute  of  Technology 
Artificial  Intelligence  Lab 
5*5  Technology  Square 
Cambridge,  MA  02139 

1  Dr.  James  A.  Paulson 
Portland  State  University 
P.0.  Box  751 
Portland,  OR  97207 

1  Dr.  Nancy  Pennington 
University  of  Chicago 
5801  S.  Ellis  Avenue 
Chicago,  IL  60637 

1  Mr.  L.  Petrullo 
2*31  N.  Edgewood  Street 
ARLINGTON.  VA  22207 

1  Dr.  Richard  A.  Poliak 
Director,  Special  Projects 
Minnesota  Educational  Computing  Consort 
2520  Broadway  Drive 
St.  Paul ,MN 

1  Dr.  Martha  Poison 
Department  of  Psychology 
Campus  Box  3*6 
University  of  Colorado 
Boulder,  CO  80309 

1  DR.  PETER  P0LSCN 
DEPT.  OF  PSYCHOLOGY 
UNIVERSITY  OF  COLORADO 
BOULDEk,  CO  80309 

1  Dr.  Fred  Reif 
Physics  Department 
University  of  California 
Berkeley ,  CA  9*720  i 
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1  Dr.  Lauren  Resnick 
LRDC 

University  of  Pittsburgh 
3939  O'Hara  Street 
Pittsburgh.  PA  1521 

1  Dr.  Jeff  Richardson 
Denver  Research  Institute 
University  of  Denver 
Denver,  CO  80208 

1  Mary  S.  Riley 
Program  in  Cognitive  Science 
Center  for  Human  Information  Processing 
University  of  California,  San  Diego 
La  Jolla,  CA  920 93 

1  Dr.  Andrew  M.  Rose 

American  Institutes  for  Research 
1055  Thomas  Jefferson  St.  NW 
Washington,  DC  20007 

1  Dr.  Ernst  Z.  Rothkopf 
Bell  Laboratories 
Murray  Hill.  NJ  0797R 

1  Dr.  William  B.  Rouse 
Georgia  Institute  of  Technology 
School  of  Industrial  &  Systems 
Engineering 
Atlanta,  GA  30332 

1  Dr.  David  Rumelhart 
denter  for  Human  Information  Processing 
Uhiv.  of  California,  San  Diego 
La  Jolla,  CA  92093 

1  Dr.  Michael  J.  Samet 
fierce ptronics,  Inc 
6fc71  Variel  Avenue 
Woodland  Hills,  CA  91369 

1  Dr.  Roger  Schank 
Yble  University 
Department  of  Computer  Science 
P.0.  Box  2158 
Nfew  Haven,  CT  06520 

1  Dr.  Walter  Schneider 
Psychology  Department 
603  E.  Daniel 
Champaign,  IL  61820 
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1  Dr.  Alan  Sehoenfeld 
Mathematics  and  Education 
The  University  of  Rochester 
Rochester,  NY  19627 

1  DR.  ROBERT  J.  SEIDEL 

INSTRUCTIONAL  TECHNOLOGY  GROUP 
HUMRRO 

300  N.  WASHINGTON  ST. 

ALEXANDRIA,  VA  22319 

1  Mr.  Colin  Sheppard 
Applied  Psychology  Unit 
Admiralty  Marine  Technology  Eat. 
Teddington,  Middlesex 
United  Kingdom 

1  Dr.  H.  Wallace  Sinaiko 
Program  Director 

Manpower  Research  and  Advisory  Services 
Smithsonian  Institution 
801  North  Pitt  Street 
Alexandria,  VA  223 Ik 

1  Dr.  Edward  E.  Smith 

Bolt  Beranek  &  Newman,  Inc. 

50  Moulton  Street 
Cambridge,  MA  02138 

1  Dr.  Richard  Snow 
School  of  Education 
Stanford  University 
Stanford,  CA  94305 

1  Dr.  Eliott  Soloway 
Yale  University 

Department  of  Computer  Science 

P.0.  Box  2158 

New  Haven,  CT  06520 

1  Dr.  Kathryn  T.  Spoehr 
Psychology  Department 
Brown  University 
Providence,  RI  02912 

1  Dr.  Robert  Sternberg 
Dept,  of  Psychology 
Yale  University 
Box  11A,  Yale  Station 
New  Haven,  CT  06520 
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1  Dr.  Albert  Stevens 

Bolt  Beranek  &  Newman,  Inc. 

10  Houlton  St. 

Cambridge,  HA  02238 

1  David  E.  Stone,  Ph.D. 

Hazeltine  Corporation 
7 680  Old  Springhouse  Road 
He Lean,  VA  22102 

1  DR.  PATRICK  SUPPES 

INSTITUTE  FOR  HATHEHATICAL  STUDIES  IN 
THE  SOCIAL  SCIENCES 
STANFORD  UNIVERSITY 
STANFORD.  CA  9*305 

1  Dr.  Klkual  Tatsuoka 

Computer  Based  Education  Research  Lab 
252  Engineering  Research  Laboratory 
Urbana.  IL  61801 

1  Dr.  Maurice  Tatsuoka 

Department  of  Educational  Psychology 
University  of  Illinois 
Urbana,  IL  61801 
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